Since the release of ArcGIS Enterprise 10.5.1, distributed collaboration has been a powerful way to connect and integrate your web GIS across a network of participants.  Distributed collaboration has made it possible to organize and share content between entities and organizations including departments, other governments, private businesses and more.

During the time since the release of distributed collaboration, the implementation of ArcGIS Enterprise “on premises” at customer sites has also grown.  And for good reason!  Deploying ArcGIS Enterprise at your site gives you complete control over your deployment while providing the core capabilities of organization-wide mapping, analysis, data management, sharing and collaboration.

While working with ArcGIS Enterprise customers I have discovered a common misconception regarding the ability of an internal-only ArcGIS Enterprise to collaborate with ArcGIS Online.  A common deployment pattern for ArcGIS Enterprise is to not deploy the system as public-facing at your site.  The system is deployed for internal customers only.  The general public cannot connect to this system over the public Internet and items, services, and data are all behind the organization’s firewall.  The misconception is that this internal-only ArcGIS Enterprise cannot sync (send and receive) items such as web maps, apps, and feature layers with ArcGIS Online.  It can!

In such a scenario, even though ArcGIS Online is considered the host of the collaboration, all communication is initiated by ArcGIS Enterprise from behind your firewall.  To enable this, organization firewall rules must be configured to support outbound communication on port 443.  A few other technical details include:

The following URLs must be whitelisted/reachable by the ArcGIS Enterprise machine:

• https://ago-item-storage.s3.amazonaws.com
• https://*.arcgis.com
• https://*.arcgisonline.com
• http://*.esri.com
• https://*.blob.core.windows.net

The Portal for ArcGIS service account that runs the Portal for ArcGIS service needs access to the Internet.

Once the above requirements are met you may proceed to set up your collaboration.

It might seem counterintuitive that your internal ArcGIS Enterprise can sync with ArcGIS Online and even receive items.  It is not only possible, but can be a very effective way to extend the reach and use of your web GIS. 

At the Architecture Practice, we are getting a lot of questions about shared instance pools.  

Before 10.7.0, the solution for reducing the amount of RAM was to set low-utilization so that the minimum number of instances in that service’s dedicated pool to zero. By doing so, you allow ArcGIS Server to not run any ArcSOCs for the service if it hasn’t received any requests in a while. 

This “min-zero” solution eliminates the resource usage for services that are going unused. Because you can still set the maximum number of instances, you can accommodate services that receive infrequent bursts of traffic.  The next time the service gets a request, an ArcSOC powers up to handle it at the cost of the startup time of the ArcSOC.  Also, this service could then sit idle for a more extended period, consuming the started SOC until the service hits the set idle timeout.

At 10.7.0, Esri announced support for shared instance pools. Every ArcGIS Server Site now comes with a shared instance pool, containing four ArcSOC processes by default. This number can increase to accommodate more services. The shared instance pool utilizes all of the SOCs assigned to it, so you should only increase the pool as you need to.

Once a compatible map service has published to your ArcGIS Server Site, you can designate it to use the shared pool. Any service added to the shared instance pool will no longer have its dedicated pool; it will dip into the shared pool and use a SOC or two as needed. Once it’s done handling a request, that ArcSOC is free to be used by any other service in the shared pool.

The following restrictions limit what services can use the shared instance pool:

• Only map services published from ArcGIS Pro can be configured to use the shared instance pool. Other service types, such as geoprocessing services, are not supported.
• Only specific capabilities of map services—feature access, WFS, WMS, and KML—can be enabled. Turn off all other capabilities before continuing.
• Services that have custom server object extensions (SOEs) or server object interceptors (SOIs) cannot use shared instances.
• Services published from ArcMap cannot use shared instances.
• Cached map services published from ArcGIS Pro that meet the above requirements can use shared instances.

For further information, please see:

Introducing shared instances in ArcGIS Server 10.7 

Configure service instance settings—ArcGIS Server Administration (Windows) | ArcGIS Enterprise 

Interest has been expressed within the Esri community with having Esri provide a set of icons for building presentations, particularly those involving conceptual IT architectures. Esri recently released an initial set of "Esri Presentation Icons" which can be used to develop conceptual architecture diagrams to provide a unified view of GIS deployments. Attached is the initial set of icons that are being provided for customer usage.

The intended use of this icon set is to enhance PowerPoint presentations by illustrating core GIS concepts. These are not intended to be used for schematic drawings or highly detailed workflows.

• 
   target="_self">Infrastructure Deployment
• Configuration Management
• Putting it together
• Summary

In this entry, we will be looking at what a deployment looks like from the infrastructure as code (IaC) perspective with Terraform as well as the configuration management side with PowerShell DSC (Desired State Configuration). Both play important roles in automating ArcGIS Enterprise deployments, so let's jump in.

This deployment will follow a single machine model as described in the ArcGIS Enterprise documentation. It will consist of the following.

• Portal for ArcGIS 10.7.1

• ArcGIS Server 10.7.1 (Set as Hosting Server)

  • Services Directory is disabled

• ArcGIS Data Store 10.7.1 (Relational Storage)

• Two (2) Web Adaptors within IIS

  • Portal context: portal

  • Server context: hosted

  • Self-Signed certificate matching the public DNS

Additional Configurations

• WebGISDR is configured to perform weekly full backups that are stored within Azure Blob Storage via Task Scheduler

• Virtual machine is configured for nightly backups to an Azure Recovery Services Vault

• RDP (3389) access is restricted via Network Security Group to the public IP of the box in which Terraform is ran from.

• Internet access (80, 443) is configured for ArcGIS Enterprise via Network Security Group

• Azure Anti-malware is configured for the virtual machine

The complete code and configurations can be found attached below. You will need to provide your own ArcGIS Enterprise licenses however.

Note:   This is post two (2) in a series on engineering with ArcGIS.

Infrastructure Deployment

If you are not already familiar with Terraform and how it can be used to efficiently handle the lifecycle of your infrastructure, I would recommend taking the time to read through the first entry in this series which can be found here. The Terraform code in that first entry will be used as the basis for the work that will be done in this posting.

As discussed in the first entry, Terraform is a tool designed to help manage the lifecycle of your infrastructure. Instead of rehashing the benefits of Terraform this time however, we will jump straight into the code and review what is being done. As mentioned above, the template from the first entry in this series is used again here with additional code added to perform specific actions needed for configuring ArcGIS Enterprise. Let's take a look at those additions.

These additions handle the creation of two blob containers that will be used for uploading deployment resources ("artifacts") and a empty container ("webgisdr") that will be used when configuring webgisdr backups along with the uploading of the license files, the PowerShell DSC archive and lastly, the web adaptor installer.

resource "azurerm_storage_container" "artifacts" {
  name                  = "${var.deployInfo["projectName"]}${var.deployInfo["environment"]}-deployment"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
  storage_account_name  = "${azurerm_storage_account.storage.name}"
  container_access_type = "private"
}

resource "azurerm_storage_container" "webgisdr" {
  name                  = "webgisdr"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
  storage_account_name  = "${azurerm_storage_account.storage.name}"
  container_access_type = "private"
}

resource "azurerm_storage_blob" "serverLicense" {
  name                   = "${var.deployInfo["serverLicenseFileName"]}"
  resource_group_name    = "${azurerm_resource_group.rg.name}"
  storage_account_name   = "${azurerm_storage_account.storage.name}"
  storage_container_name = "${azurerm_storage_container.artifacts.name}"
  type                   = "block"
  source                 = "./${var.deployInfo["serverLicenseFileName"]}"
}

resource "azurerm_storage_blob" "portalLicense" {
  name                   = "${var.deployInfo["portalLicenseFileName"]}"
  resource_group_name    = "${azurerm_resource_group.rg.name}"
  storage_account_name   = "${azurerm_storage_account.storage.name}"
  storage_container_name = "${azurerm_storage_container.artifacts.name}"
  type                   = "block"
  source                 = "./${var.deployInfo["portalLicenseFileName"]}"
}

resource "azurerm_storage_blob" "dscResources" {
  name                   = "dsc.zip"
  resource_group_name    = "${azurerm_resource_group.rg.name}"
  storage_account_name   = "${azurerm_storage_account.storage.name}"
  storage_container_name = "${azurerm_storage_container.artifacts.name}"
  type                   = "block"
  source                 = "./dsc.zip"
}

resource "azurerm_storage_blob" "webAdaptorInstaller" {
  name                   = "${var.deployInfo["marketplaceImageVersion"]}-iiswebadaptor.exe"
  resource_group_name    = "${azurerm_resource_group.rg.name}"
  storage_account_name   = "${azurerm_storage_account.storage.name}"
  storage_container_name = "${azurerm_storage_container.artifacts.name}"
  type                   = "block"
  source                 = "./${var.deployInfo["marketplaceImageVersion"]}-iiswebadaptor.exe"
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This addition handles the generation of a short-lived SAS token from the storage account that is then used during the configuration management portion to actually grab the needed files from storage securely. In this situation, we could simplify the deployment by marking our containers as public and not requiring a token but that is not recommended.

data "azurerm_storage_account_sas" "token" {
  connection_string = "${azurerm_storage_account.storage.primary_connection_string}"
  https_only        = true
  start             = "${timestamp()}"
  expiry            = "${timeadd(timestamp(), "5h")}"

  resource_types {
    service   = false
    container = false
    object    = true
  }

  services {
    blob  = true
    queue = false
    table = false
    file  = false
  }

  permissions {
    read    = true
    write   = true
    delete  = true
    list    = true
    add     = true
    create  = true
    update  = true
    process = true
  }
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The final change is the addition of an extension to the virtual machine that will handle the configuration management task using PowerShell DSC. Instead of reviewing this in-depth here, just know that the data that is getting passed under the settings and protected_settings json will be passed to PowerShell DSC as parameters for use as needed by the configuration file.

resource "azurerm_virtual_machine_extension" "arcgisEnterprise-dsc" {
  name                       = "dsc"
  location                   = "${azurerm_resource_group.rg.location}"
  resource_group_name        = "${azurerm_resource_group.rg.name}"
  virtual_machine_name       = "${element(azurerm_virtual_machine.arcgisEnterprise.*.name, count.index)}"
  publisher                  = "Microsoft.Powershell"
  type                       = "DSC"
  type_handler_version       = "2.9"
  auto_upgrade_minor_version = true
  count                      = "${var.arcgisEnterpriseSpecs["count"]}"

  settings = <<SETTINGS
	{
		"configuration": {
          "url": "${azurerm_storage_blob.dscResources.url}${data.azurerm_storage_account_sas.token.sas}",
          "function": "enterprise",
		  "script": "enterprise.ps1"
		},
		"configurationArguments": {
          "webAdaptorUrl": "${azurerm_storage_blob.webAdaptorInstaller.url}${data.azurerm_storage_account_sas.token.sas}",
          "serverLicenseUrl": "${azurerm_storage_blob.serverLicense.url}${data.azurerm_storage_account_sas.token.sas}",
          "portalLicenseUrl": "${azurerm_storage_blob.portalLicense.url}${data.azurerm_storage_account_sas.token.sas}",
          "externalDNS": "${azurerm_public_ip.arcgisEnterprise.fqdn}",
		  "arcgisVersion" : "${var.deployInfo["marketplaceImageVersion"]}",
          "BlobStorageAccountName": "${azurerm_storage_account.storage.name}",
          "BlobContainerName": "${azurerm_storage_container.webgisdr.name}",
          "BlobStorageKey": "${azurerm_storage_account.storage.primary_access_key}"
      }
	}
	SETTINGS
  protected_settings = <<PROTECTED_SETTINGS
	{
		"configurationArguments": {
          "serviceAccountCredential": {
            "username": "${var.deployInfo["serviceAccountUsername"]}",
            "password": "${var.deployInfo["serviceAccountPassword"]}"
      },
		"arcgisAdminCredential": {
			"username": "${var.deployInfo["arcgisAdminUsername"]}",
			"password": "${var.deployInfo["arcgisAdminPassword"]}"
			}
		}
	}
	PROTECTED_SETTINGS
}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Configuration Management

As was touched on above, we are utilizing PowerShell DSC (Desired State Configuration) to handle the configuration of ArcGIS Enterprise as well as a few other tasks on the instance. To simplify things, I have included v2.1 of the ArcGIS module within the archive but the public repo can be found here. The ArcGIS module provides a means with which to interact with ArcGIS Enterprise in a controlled manner by provided various "resources" that perform specific tasks. One of the major benefits of PowerShell DSC is that it is idempotent. This means that we can continually run our configuration and nothing will be modified if the system matches our code. This provides administrators the ability to push changes and updates without altering existing resources as well as detecting configuration drift over time. 

To highlight the use of one of these resources, let's take a quick look at the ArcGIS_Portal resource which is designed to  configure a new site without having to manually do so through the typical browser based workflow. In this deployment, our ArcGIS_Portal resource looks exactly like the below code. The resource specifies the parameters that we must be provided to successfully configure the portal site and will error out if all required parameters are not provided. 

ArcGIS_Portal arcgisPortal {
    PortalEndPoint = (Get-FQDN $env:COMPUTERNAME)
    PortalContext = 'portal'
    ExternalDNSName = $externalDNS
    Ensure = 'Present'
    PortalAdministrator = $arcgisAdminCredential
    AdminEMail = 'example@esri.com' 
    AdminSecurityQuestionIndex = '12'
    AdminSecurityAnswer = 'none'
    ContentDirectoryLocation = $portalContentLocation
    LicenseFilePath = (Join-Path $(Get-Location).Path (Get-FileNameFromUrl $portalLicenseUrl))
    DependsOn = $Depends
}
$Depends += '[ArcGIS_Portal]arcgisPortal'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Because of the scope of what is being done within the configuration script here, we will not be doing a deep dive. This will come in a later article.

Putting it together

With the changes to the Terraform template as well as a very high level overview of PowerShell DSC and its purpose, we can deploy the environment using the same commands mentioned in the first entry in the series. Within the terminal of your choosing, navigate into the extracted archive that contains your licenses, template and DSC archive, and start by initializing Terraform with the following command.

terraform init

Next, you can run the following to start the deployment process. Keep in mind, we are not only deploying the infrastructure but configuring ArcGIS Enterprise so the time for completion will vary. When it completes, it will output the public facing url to access your ArcGIS Enterprise portal.

terraform apply

Summary

As you should quickly be able to see, removing the manual configuration aspects of software as well as the deployment of infrastructure, a large portion of problems can be mitigated by moving toward IaC and Configuration Management. There are many solutions out there to handle both aspects and these are just two options. Explore what works for you and start moving toward a more DevOps centric approach.

I hope you find this helpful, do not hesitate to post your questions here: Engineering ArcGIS Series: Tools of an Engineer 

Note: The contents presented above are examples and should be reviewed and modified as needed for each specific environment.

It’s a long-time dilemma for many organizations whether to devote resources to custom software application development or go for the commercial off-the-shelf (COTS) solution? In this blog post, we will highlight some main fundamental strategies that will help any GIS department choose what will best fit its user needs.

In short, the answer to this question also links us back to what the requirements are and what current infrastructure is. An application implementation strategy is an approach to delivering capabilities that meet your business needs with technology.

Gathering Requirements is crucial for the success of your application whether it's COTS or custom. You should closely work with the sponsor, stakeholders, users, and IT focusing on the requirements, not the solution. Types of the requirements you will have to gather are:

Following the requirements, there are many other factors to consider when deciding the best way to deliver new capabilities through apps. These factors include resourcing, initial development effort, ongoing app maintenance, user training, and technical support. In addition, users now expect frequent updates to their apps, which increases demand for resources to develop and maintain custom apps. As a result, it’s best to select the approach that delivers the capabilities you need with the least cost and effort. An ideal strategy will minimize cost and optimize the use of development resources.

By applying a “configure first” philosophy that prioritizes commercial off-the-shelf (COTS) apps and least-effort design patterns, you can reduce the cost and effort needed to deploy and maintain applications for your users. Organizations that adopt a configure-first philosophy start by configuring COTS apps, then extend and customize apps only when needed. Using this least-effort approach in your application implementation strategy lets you deliver capabilities faster and reserve your development resources for more complex tasks.

Depending on your specific requirements, you can:

• Configure COTS apps to meet your business needs. ArcGIS provides many configurable COTS apps that support key workflows out of the box. Using COTS apps requires the least effort and the lowest ongoing cost. ArcGIS COTS Apps (Web App Builder, Story Maps, Operational Dashboards, Collector, GeoForm, and Survey 123)

• Extend existing apps, either by modifying templates or by creating widgets for COTS apps. Esri offers app templates at solutions.arcgis.com and github.com/esri that provide focused solutions for specific problems; you can modify the source code for these templates to add discrete capabilities. In addition, several ArcGIS COTS apps use modular frameworks that let you create custom widgets and plug them into the apps. Extending existing apps lets you develop only the additional functionality you need, saving money and effort.

• Customize apps using ArcGIS APIs and SDKs. These APIs and SDKs provide objects like the Identity Manager to manage credentials within custom apps that expose parts of the ArcGIS platform (such as secure web maps). Because you don’t have to code those parts yourself, you can build business-focused apps to take advantage of ArcGIS COTS capabilities, reducing the overhead for app development and maintenance. Check out developers.esri.com for more information on the wide selection of APIs and SDKs available.

In conclusion, to establish an effective application implementation strategy for your organization, look deep into your requirements and available resources, and try to follow these 3 simple guidelines:

1. Adopt a configure-first philosophy, configuring COTS apps when possible to deliver the capabilities you need.
2. If you have a requirement that cannot be met with configuration alone, extend existing apps with discrete capabilities and widgets.
3. When you need capabilities that you can’t provide by configuring and extending existing apps, customize apps using ArcGIS APIs and SDKs.

Numerous technical sessions, spotlight talks, and conversations referenced the "Architecting the ArcGIS Platform: Best Practices" whitepaper (https://go.esri.com/bp). This document presents some implementation guidelines in the form of a conceptual reference architecture diagram and associated best practice briefs. You can use these guidelines to maximize the value of your ArcGIS implementation and meet your organizational goals.

GIS has evolved at a rapid pace ever since computers took up to the challenge of providing spatial capabilities. The evolution of GIS from a Map creation platform to a Location Intelligence platform necessitates the need to build systems that are robust, reliable, and elastic as they are utilized to host solutions that your business relies on to be successful. GIS is everywhere running from servers running within traditional data centers, the Cloud, and from mobile and IoT devices. Supporting such a diverse landscape creates unique architectural challenges that require a systematic approach to designing your GIS.

Esri system architecture design is based upon traditional architecture patterns centered around multiple tiers, namely a Client/Application, Presentation, Services, Support, and Data tier. Each tier aligns with Esri products and solution components as depicted by the example logical architecture below. Following a systematic approach, this blog series will explore the various architectural tiers and their related solution components in support of building modern GIS platforms to meet today's business and Location Intelligence needs.

• What is Infrastructure as Code?
• What is Terraform?
• Resources
• Providers
• State
• Infrastructure Life-cycle
• Creation
• Decommission

What is Infrastructure as Code?

Taken directly from Microsoft, 

"Infrastructure as Code (IaC) is the management of infrastructure (networks, virtual machines, load balancers, and connection topology) in a descriptive model".

In the simplest terms, Infrastructure as Code (IaC) is a methodology to begin treating cloud infrastructure the same as application source code. No longer are changes made directly to the infrastructure, but to the source code for a given environment or deployment. This change in behavior will lead to environments that are easily reproducible, quickly deployable and accurate.

Note:   This post is one (1) in a series on engineering with ArcGIS.

What is Terraform?

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage components such as virtual machines, networking, storage, firewall rules and many others. To define what needs to be deployed or changed, terraform uses what is called a terraform configuration which can be made up of one or more individual files within the same folder. Terraform files utilize the file extension

.tf

as well as HCL (HashiCorp Configuration Language) as the configuration language. HCL is a structured configuration language that is both human and machine friendly for use with command-line tools, but specifically targeted towards devops tools, servers, etc. 

In complex deployments, administrators may find they can utilize a different configuration file for each aspect of their environment as such,

/environment_a

   /production

      network.tf

      machines.tf

      storage.tf

      security.tf

   /staging

      network.tf

      machines.tf

      storage.tf

      security.tf

whereas with simple deployments, a single file may be sufficient, such as

/environment_a

   main.tf

To actually create and manage infrastructure, terraform has a number of constructs to allow users to define Infrastructure as Code but the most important two are Providers and Resources. 

Resources

Resources are the mechanism that tell terraform how the infrastructure should be deployed and configured. Each cloud provider will have its own list of Resources that users will have access to. An example would look something like this which creates an azure resource group and a virtual network within.

resource "azurerm_resource_group" "rg" {
   name = "prd"
   location = "westus2"

}

resource "azurerm_virtual_network" "vnet" {
   name = "prd-vnet"
   location = "westus2"
   address_space = ["10.0.0.0/16"]
   resource_group_name = "prd"
}

Providers

Providers are the mechanism for defining what cloud provider or on-premise Resources are available for use. Each provider offers a set resources, and defines for each resource which arguments it accepts, which attributes it exports, and how changes to resources of that type are actually applied to remote APIs. Most of the available providers correspond to one cloud or on-premises infrastructure platform, and offer resource types that correspond to each of the features of that platform. In simpler terms, if the goal is to define infrastructure on Azure, an Azure based provider must be used before Resources can be defined.

A provider example for Azure would look something like this,

provider "azurerm" {
   subscription_id = "this-is-not-a-real-subscription-id"
   client_id = "this-is-not-a-real-client-id"
   client_secret = "this-is-not-a-real-client-secret"
   tenant_id = "this-is-not-a-real-tenant-id"
}

Terraform has multiple methods for authenticating to a given cloud provider and in this example, a Service Principal is being utilized.

State

Lastly, terraform makes use of a State File that keeps track of the infrastructure that has been deployed and configured. This state file is what allows terraform to run checks against the last recorded state of an environment compared to the current run and provide users the delta so validation can be done before making changes. This aspect is very important in that it allows terraform to be idempotent, which is a key aspect of IaC.

Infrastructure Life-cycle

For the purposes of this introduction, we will be using the attached terraform template as the basis for the following examples. This template will be posted to GitHub in the near future for future articles to utilize and build off. It is designed to deploy the following within an Azure subscription.

Before it can be deployed, the deployInfo variable group (Map) will need to be populated. When creating the Service Principal, the following documentation (Service Principal creation) can be reference for assistance. Terraform will also need to be available locally. The steps to ensure it is can be found here.

• Resource Group
• Virtual Network
• Subnet
• Network Security Group
  • Rule to allow 80 and 443 traffic from the internet into the virtual network
  • Rule to allow RDP access (3389) into the virtual network from the public IP of the person who deploys the template. This is accomplished by querying the web during deployment and retrieving your public IP. This workflow is not recommended in production environments and is only being used for example purposes.
  • Rule to block all other internet traffic into the virtual network.
• Storage account
• Availability Set
• Public IP
• Network Interface
• Virtual Machine
  • Windows Defender Extension
  • Recovery Services Vault Extension
• Key Vault
• Recovery Services Vault
  • Backup Policy

Creation

Once terraform is available locally and the deployInfo variable map has been completed, the first step in deploying infrastructure is to initialize terraform. This can be accomplished by navigating to the directory in which you have saved the above template with the .tf file extension and running the following command, which will prepare various local settings and data that will be used by subsequent commands.

terraform init

The output from initializing terraform will resemble the following.

With terraform successfully initialized, the next step in the process is to have terraform review the template and determine what changes need to take place. This step will have terraform compare the template with the current state file and produce an output showing the deltas as follows. To do so, use the following command.

terraform plan

Output has been truncated.

As this is the first run of terraform, terraform is only able to see resources it needs to add (create). Later in this post, we will walk through updating existing resources.

With terraform successfully prepared to deploy our infrastructure, we can being the deployment by using the following command which will start the process of creating the resources within Azure and provide the following output when complete.

terraform apply

Updates

As the deployment is utilized, it may be determined that the current infrastructure is not adequately sized and resources need to be increased. As our resources are defined as code, so is the virtual machine sizing. Within the "arcgisEnterpriseSpecs" variable map, a variable is defined with the name "size" which is the Azure machine sizing that is used by terraform when deploying resources.

If this variable is changed to "Standard_D8s_v3" and terraform plan is ran again, it will detect a delta between what is defined in code and what is actually deployed within Azure and present this in the output as such.

Once this delta is planned and the changes are ready to be pushed to the actual infrastructure within Azure, simply running terraform apply again will begin the process.

Decommission

The last phase of a given deployments life-cycle is decommissioning. Once infrastructure has reached the end of it life, it must be terminated and removed and thankfully, terraform provides an easy method to do so with the following command.

terraform destroy

The output of destroying the infrastructure will resemble the following.

In conclusion, as teams aim to become more agile and move at a much faster pace, moving to modern methodologies such as infrastructure as code (IaC) is a great first step and should not be viewed as unnecessary but a crucial step in the right direction.

I hope you find this helpful, do not hesitate to post your questions here: https://community.esri.com/community/implementing-arcgis/blog/2019/07/03/engineering-arcgis-series-t... 

Note: The contents presented above are examples and should be reviewed and modified as needed for each specific environment.